Map view

ABSTRACT

One or more systems, devices, and/or methods for providing navigation-related functions, such as map display, are disclosed. For example, a method may include receiving input corresponding to a location. Photo image data associated with the location are retrieved. The photo image data are based on at least one image captured by a video camera or a photo camera. The photo image data are provided to display at least a portion of a photo image view representing at least a portion of the location. Map data corresponding to the location are retrieved. The map data are provided to display at least a portion of a digital map in the portion of the photo image view, and the digital map is integrated within the photo image view.

BACKGROUND OF THE INVENTION

The present invention relates to imagery, maps, and/or navigation and,more particularly, to providing imagery with maps or map data, such asfor geographic and/or navigation functions.

Navigation and/or geographic systems or data thereof are used to helppeople travel or explore the world around them. For example, map viewsor imagery of different areas may provide useful guidance orgeographic/location information. For example, one can view a map orgeographical imagery to find his or her way to a destination or to viewa desired point-of-interest (“POI”).

As systems, devices, technologies, and/or user interfaces advance,people are given more options on how to view the world around them.However, technical limitations and inefficiencies may still exist.Accordingly, improvements and unique ways to provide and displayrepresentations of areas to users are desired.

SUMMARY OF THE INVENTION

According to one aspect, a method for providing navigation-relatedfunctions, such as map display, is provided. For example, the methodincludes receiving input corresponding to a location. Photo image dataassociated with the location are retrieved. The photo image data arebased on at least one image captured by a video camera or a photocamera. The photo image data are provided to display at least a portionof a photo image view representing at least a portion of the location.Map data corresponding to the location are retrieved. The map data areprovided to display at least a portion of a digital map in the portionof the photo image view, and the digital map is integrated within thephoto image view.

The present invention is defined by the following claims, and nothing inthis section should be taken as a limitation on those claims. Furtheraspects and advantages of the invention are discussed below inconjunction with the preferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an exemplary map, geographic, and/or navigationdata system.

FIG. 2 illustrates an embodiment of one or more devices that may be usedin the system of FIG. 1.

FIG. 3 illustrates components of an image view and/or associatedprocessing, such as used in the system of FIG. 1.

FIG. 4 illustrates an exemplary communication system and/or devicesthereof related to the system of FIG. 1 or products thereof.

FIG. 5 illustrates a graphics user interface on a device, such as adevice in the system of FIG. 4.

FIG. 6 illustrates a diagram regarding different views or viewpoints.

FIG. 7 is a flowchart of an exemplary method of generating data for amap, geographic, and/or navigation device or system.

FIG. 8 is a flowchart of an exemplary method of operating a map,geographic, and/or navigation device or system.

FIG. 9 illustrates an exemplary image view via a graphics userinterface, such as the graphics user interface of FIG. 5.

DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS

FIG. 1 shows one embodiment of a geographic, map, and/or navigation datasystem 100. The geographic, map, or navigation data system 100 includes,but is not limited to, an object 108 travelling on a path network 104 ina geographic region 102, a device 112, a processing device 120, aworkstation or computer 124, and/or a database 128. Additional, fewer,or different components may be provided. For example, a proxy server, aname server, a map server, a cache server or cache network, a router, aswitch or intelligent switch, additional computers or workstations, auser interface, administrative components, such as an administrativeworkstation, a gateway device, a backbone, ports, transmitter/receivercomponents, a network, network connections, and/or network interfacesmay be provided. While the components in FIG. 1 are shown as separatefrom one another, one or more of these components may be combined. Also,a combined component may be separated into multiple components.

The object 108 is a vehicle, such as a car, truck, motorcycle, bicycle,Segway, or other transportation device or mechanical device used to moveon the path network 104. Alternatively, the object 108 may represent apedestrian, such as a human being or animal, that walks or moves alongthe path network 104. The path network 104 is a road network and/or acollection of other pathways. For example, the path network 104 is aroad network made up of various roads. The roads may be used to drivevehicles on, such as local and neighborhood streets as well as highways.Also, instead of or in addition to traditional streets, the path network104 may include bicycle roads or paths, walking paths, or other travelpaths. Alternatively, the path network 104 may be an open area spacewith no specific, dedicated, or predetermined paths or travelconstraints. The path network 104 is in the geographic region 102, suchas a city, a suburb, a state, a country, and/or other geographic region.

The object 108 travels along or moves about the path network 104 tocollect data. A supporting device or housing 112 is attached orconnected with or carried by the object 108. The supporting device 112may be or may include equipment to collect data representing an areaabout or around a road or path or other area. For example, the collecteddata may be imagery data, video/camera data (such as in the visiblespectrum or other spectrum), laser data, light data (such as LightDetection and Ranging (“LIDAR”) data), and/or other visual related orlight based data. Other data such as location data, global positioningsystem (“GPS”) data, and/or other geographic or position data may becollected.

In one embodiment, the supporting device 112 is a housing or containerthat is attached (such as removably attached or integrally attached) toor carried by the object 108. As the object 108 is on and/or moving onthe path network 104 or portions thereof, equipment, such as a camerasystem and/or LIDAR device, in or on the supporting device 112 collectsor gathers data corresponding to the surrounding area. Alternatively,the supporting device 112 may be one or more camera devices, LIDARdevices, and/or other devices or parts thereof. The supporting device112 may be positioned at a top backend of the object 108 (such as avehicle) and may be angled to enhance collection. In other embodiments,the supporting device 112 and/or equipment thereof may be positioned inany one or more places on or in the object 108 in any direction.

FIG. 2 illustrates an embodiment of components of a device 201. Forexample, the device 201 may be an embodiment of the object 108 (such asa vehicle), an embodiment of the supporting device 112, or combinationsthereof. In one embodiment, the device 201 includes, but is not limitedto, a processor 209, a memory 213, a positioning system 221, datacollection device(s) 217, one or more cameras or camera system 205, andan input device 225. Additional, fewer, or different components may beprovided. For example, transmitter components, receiver components,network components, and/or other components may be provided. While thecomponents in FIG. 2 are shown as separate from one another, one or moreof these components may be combined. Also, some of the components may beprovided outside of the device 201.

The processor 209 is a general processor, application-specificintegrated circuit (“ASIC”), digital signal processor, fieldprogrammable gate array (“FPGA”), digital circuit, analog circuit,and/or combinations thereof. In one embodiment, the processor 209 is oneor more processors operable to control and/or communicate with thevarious electronics and logic of the device 201. The memory 213 is anyknown or future storage device. The memory 213 is a non-volatile and/orvolatile memory, such as a Random Access Memory “RAM” (electronic), aRead-Only Memory “ROM” (electronic), or an Erasable ProgrammableRead-Only Memory (EPROM or Flash memory). A memory network may beprovided. The memory 213 may be part of the processor 209. The memory213 is operable or configured to store image data, video data, and/orother data, such as collected data.

The processor 209 or other processor is configured or adapted to executedata collection applications or other applications, which may be storedin the memory 213 or other memory. For example, the processor 209communicates with the other components in the device 201 and managescollection of the various data, such as image and/or video data. Forexample, synchronization (such as with location data) and storageprocesses may be executed by the processor 209 or other processors.

The positioning system 221 is one or more Global Positioning System(“GPS”) devices, one or more mechanical and/or electrical sensors, oneor more gyroscopes, a local positioning system, one or more directionsensors, or other system or device(s) used for providing position data,such as location data (e.g., longitude, latitude, and/or altitude or anyother coordinates) and/or direction data, of the device 201 and/orcomponents thereof.

The camera system 205 is one or more cameras used for taking picturesand/or videos of a surrounding area. For example, the camera system 205includes one or more video cameras that record video data (such as inthe visible light spectrum) representing geographic and/or man-madefeatures of and about a road or path as the device 201 (such as theobject 108 and/or the device 112) moves along a road, a path, or an openarea. The camera system 205 may also capture still photographs separatefrom the video data (and/or video data may be used to provide stillphotographs or images). The camera system 205 is able to capturedifferent colors and associated text of different features. In oneembodiment, multiple cameras face different directions. For example, oneor more forward facing cameras, one or more right side facing cameras,one or more left side facing cameras, and one or more backward facingcameras relative to the device 201 (such as a vehicle or harness) areprovided. In one embodiment, four, six, eight, or any number of camerasare provided. In addition to or instead of static position cameras, oneor more rotating cameras may also be provided. For example, a camera(such as a panoramic camera) that rotates 360 degrees or less may beprovided, such as to collect multiple image frames. Some or all of thecameras may be fitted with wide-angle lenses (such as a fisheye lens)that provide angle of views that are greater than the human eye, such asto capture panoramic images and/or video.

The data collection device(s) 217 may include one or more light datacollection devices or other data collection devices. For example thedevice or system 217 may include one or more light sources and one ormore light receivers. In one embodiment, the data collection device 217is a Light Detection and Ranging (“LIDAR”) device or sensor, a laserdevice, and/or other device that collects data points, such asthree-dimensional data, by transmitting and receiving light. Forexample, the data collection device 217 is a LIDAR device that uses oneor more lasers to collect data points representing a surrounding area,such as an area about a road or path or other area. The LIDAR devicecollects and gathers data points in a point cloud, such as a threedimensional (“3D”) point cloud, in which each data point corresponds toa local coordinate, such as (x, y, z). The one or more lasers may be ina near infrared spectrum (such as about 700 nm to about 5000 nm or about800 nm to about 2500 nm) or other light spectrum. In one embodiment, theLIDAR device is a 64 element LIDAR sensor HDL-64E from Velodyne, Inc.,located in Morgan Hill, Calif. and/or a LMS200 series LIDAR sensor fromSICK AG, located in Waldkirch, Germany.

The input device 225 may be one or more buttons, keypad, keyboard,mouse, stylist pen, trackball, rocker switch, touch pad, voicerecognition circuit, or other device or component for controlling orinputting data in one or more of the components of the device 201.

Referring back to FIG. 1, the collected data may be stored on one ormore computer-readable media 116, such as a CD-ROM, DVD, flash drive,hard drive, or other non-transitory tangible media suitable to storedata. Alternatively, the media 116 may be signals carrying or havingdata. Separate media 116 may be used to store separate or differenttypes of data. In one embodiment, photographs (such as digital orelectronic photographs), video images, LIDAR or laser data, locationdata, and/or other geographic data collected by the object 108 and/orthe device 112 are stored in one or more media 116. The collectedimage/video data may represent areas or regions about or around a path,road, or other area. For example, the collected image or video data mayinclude geographic features, such as sky features, terrain orsurrounding features, roads or paths (such as sidewalks), road or pathmarkings (such as cross-walks or lane markings), road or path signs,points-of-interest (“POIs”) such as buildings, parks, museums, etc.,and/or other man-made and/or natural features or objects.

The collected image data and/or other collected data are sent, such asvia one or more media 116, to a map, geographic, and/or navigation datadeveloper 118, such as NAVTEQ North America, LLC located in Chicago,Ill. For example, a medium 116 may be mailed to or brought to the mapdeveloper 118. Alternatively, some or all of the collected data may betransmitted to the map developer 118 via a wireless and/or wirednetwork. For example, the network may include the Internet, an intranet,a local area network (“LAN”), a wide area network (“WAN”), a virtualprivate network (“VPN”), a server network, a cellular network, asatellite network, a broadcasting network, a wireless or wiredconnection, and/or any known or future network or connection. The term“map developer” may also include third-party contractors.

The processing device 120, maintained by the map developer 118, receivesthe collected data. For example, the processing device 120 is one ormore servers, computers (such as a desktop tower or a laptop processingunit), processors, and/or other electronic processing systems ordevices. The processing device 120 includes, but is not limited to, aprocessor 140, a memory 144, and an image software application 150.Additional, fewer, or different components may be provided.

The processing device 120 may include or be in communication with one ormore of the workstations or computers 124. For example, the workstation124 is a user interface, electronic console, and/or computer with one ormore input devices that may be used to access, control, and/orcommunicate with the processing device 120 or components thereof.

The processor 140 is similar to or different than the processor 209, andthe memory is similar to or different than the memory 213. The processor140 is configured or adapted to execute the image software application150, which may be stored in the memory 144 or other memory. For example,the memory 144 stores the received collected data, such as collectedimages/video and/or location/position data. The application 150 takesthe stored images and/or video and creates image views based on commandsand management of the processor 140.

FIG. 3 illustrates an embodiment of an image view 300. For example,received images and/or video is processed to form or generate the imageview 300, such as a photo image view and/or a panoramic image view. Theimage view 300 includes, but is not limited to, an image data section304 and a region 308. The region 308 is positioned at a bottom area ofthe region 304, but the region 308 may be positioned at or correspond toother locations within region 304. The image data region 304 correspondsto one or more images or image scenes, such as panoramic imagery. Forexample, image data collected and received are processed or transformedto generate panoramic views, such as street views or other geographicalor area views. The image data and/or imagery views may be pixilatedraster data in a bitmap, JPEG, GIF, and/or other image format. In oneembodiment, omnidirectional images (“ODIs”) are converted to panoramicor perspective images, such as by using known and future techniques.Images or image data may be stitched together or processed to providepanoramic or perspective views, such as seamless imagery that has anangle-of-view that covers about or at least more than 160 degrees or 180degrees from a reference point (such as from a horizontal and/orvertical plane).

The region 308 may correspond to a ground area or region associated withthe image data section 304. Also, the region 308 may correspond to anunfocused or blurry region of the image data 304. For example, whenstitching images or processing images to create panoramic views (such aswhen converting omnidirectional image data), lower image quality occursat the bottom or the top of panoramas, causing a blurry or unfocusedeffect. Also, the nature of collecting imagery or video while moving(such as collecting images/video via camera(s) on a vehicle) may resultin portions of images to be unfocused, blurred, and/or blocked due tothe position of the camera(s) as well as the angle of view of thelenses. In another embodiment, the region 308 may correspond to ablocked region, such as a portion of the object 108 (such as a vehicle)may be blocking the road or other geographic/area feature. For example,when a camera is mounted on a vehicle for collection purposes, the topof the vehicle or other parts of the vehicle may get in the way on theimages/video. Alternatively, the region 308 may correspond to any otherarea represented by the image data 304.

Accordingly, to utilize the unfocused, blurry, and/or blocked region308, the application 150 may link and/or integrate map data or a mapdata model or representation 312 with the region 308. For example, themap data 312 may correspond to a digital interactive map model orrepresentation corresponding to the image data 304. The map data 312 orrepresentations thereof may be processed or retrieved from theprocessing device 120 (such as part of the received collected data)and/or may be retrieved from the database 128. The data associationsbetween the image data 304 and the map data 312 are made. Also, locationdata 316, such as GPS data collected, that corresponds to thelocation(s) of the image view 300 and/or the associated map data 312 arelinked or associated with the image view 300. The processing to generateone or more image views 300 and the associated map, location data,and/or other connections/links may be automated and/or may includemanual editing or processing, such as via the workstation 124.

Referring back to FIG. 1, after one or more image views 300, associatedmap data 312 or links thereof, and/or other links are generated, they orportions/components thereof are stored in the database 128 or otherdatabase. Other collected or processed data, such as geographic/areadata, may also be stored in the database 128 or other database. Thedatabase 128 is a navigation, map, and/or geographic database and mayinclude one or more specific or categorized databases. For example, thedatabase 128 includes an image database 132 and a map database 136.More, fewer, or different components may be provided.

In one embodiment, one or more processed or generated image/video views154, such as the image view 300, are stored in the image database 132.The image view data 154 may include image data files that correspond tocomposite or stitched images that represent panoramic image/video views.Alternatively, raw, preprocessed, and/or other image/video data may bestored in the image database 132. Also, static map data that areintegrated within image views 154 may be stored in the image database132 as part of the same data file. Also, links, IDs, or dataassociations that associate map data or map representations withspecific image views may be part of the image view data 154, may bestored in the indexes 158, or may be stored somewhere else, such as inthe map database 136 or other database. The indexes 158 may be one ormore look-up tables, organization charts, links or association files orIDs, and/or any index data files that associates different data or datafiles.

The map database 136 may be part of or separate from the image database132. In one embodiment, the map database 136 includes data used fornavigation-related and/or geographic-related services. For example, thegeographic/map database 136 contains road segment/link data records 162and node data records 166 that represent a road network, such as thepath network 104. A node represents an end point of a road segmentand/or intersection points (e.g., a segment is associated with twonodes, one at each respective endpoint). The road segments and/or nodesmay be associated with attributes, such as geographic coordinates,street names, address ranges, speed limits, turn restrictions atintersections, and other vehicle navigation attributes, as well as POIdata records 170, such as gasoline stations, hotels, restaurants,museums, stadiums, offices or objects therein, automobile dealerships,auto repair shops, landmarks, indoor points/features/structures,buildings, stores, etc. Alternatively or additionally, the map database136 contains path segment/link and node data records or other data thatrepresent pedestrian paths and/or indoor areas in addition to or insteadof vehicle road record data. Also, the map database 136 may includeother map data records 174 as well as one or more indexes 178 to link,associate, and/or organize different types of data. The index records178 may be similar to or different than the index records 158. The mapdata records may be generated or created and/or updated based on thecollected geographic data (such as data collected by the object 108 ordevice 112, data in the media 116, and/or data processed or received bythe device 120) or other collected, gathered, and/or acquired data.

Further description and details on node data records, segment datarecords, and/or other navigation data records or associated data may beprovided in the U.S. Pat. No. 6,038,559, entitled SEGMENT AGGREGATION INA GEOGRAPHIC DATABASE AND METHODS FOR USE THEREOF IN A NAVIGATIONAPPLICATION, which is hereby incorporated by reference.

The map database 136 and/or the database 128 or data thereof may be amaster geographic/map database that is stored in a format thatfacilitates updating, maintenance, and development. For example, themaster geographic/map database or data in the master geographic/mapdatabase is in an Oracle spatial format, such as for development orproduction purposes. The Oracle spatial format or development/productiondatabase may be compiled into a delivery format, such as a GDF format.

The data in the production and/or delivery formats may be compiled orfurther compiled to form one or more geographic/map database products ordatabases 184, which may be used in end user navigation devices orsystems or other systems or devices. The products 184 may include bothmap data and image data, or the map data may be provided separately fromthe image/video data. Also, any corresponding link or association databetween images/video and map data or other data may be provided in theproduct 184 or other product. For example, a customer of the mapdeveloper 118 (such as a developer of navigation devices or systems, asoftware application provider, a data content provider, Internetservices provider, or other entity) may receive generated image views(such as the image view 300) and/or associated data as well as otherdata (such as map data) in a delivery format and then compile and/orconfigure such data for end user devices or end user services. Or, themap developer 118 may do the compiling. In one embodiment, geographic,location, map, and/or image data is compiled (such as into a PSF format)to organize and/or configure the data (e.g., segment/link, node,image/video views, and/or other data or attributes) for performingnavigation-related functions and/or services such as route calculation,route guidance, map display, image/video display, speed calculation,distance and travel time functions, and other functions, by a navigationdevice or other electronic device or system. The navigation-relatedfunctions may correspond to vehicle navigation, pedestrian navigation,or other types of navigation as well as geographic/map features andservices. In an alternative embodiment, raw collected image data is sentfrom the map developer 118 to another entity, and the other entityprocesses the image data to generate the image views, such as the imageview 300 or various street views or other views. The other entity mayassociate and/or integrate map data or map representations with theimage views.

FIG. 4 illustrates an exemplary communication system 401. The system 401includes, but is not limited to, a device 405, a network 413, and a userdevice 409. More, fewer, or different components may be provided. In oneembodiment, the device 405 is a server or other dataprocessing/providing device (e.g., a server maintained by the mapdeveloper 118 or other entity). The server 405 includes a processor 417and a memory 421. More, fewer, or different components may be provided.For example, other hardware and/or software features commonly associatedwith a server may be provided.

The processor 417 is similar to or different than the processor 209 or140, and the memory 421 is similar to or different than the memory 213or 144. The processor 417 is configured or adapted to control and/orcommunicate with other components, such as the memory 421. The memory421 includes a database 425 and a software application 429. Theapplication 429 may be part of or communicates with a servicesapplication that provides map and/or navigation related services, suchas image/video and map display. The database 425 may be similar to thedatabase 128, the database 132, the database 136, and/or the database184. The server 405 (via the processor 417, the memory 421, theapplication 429, and/or other components) uses the database 425 or otherdatabases to provide image/video views, such as the image view 300, torequesting users or clients, such as via the network 413.

The network 413 may include the Internet, an intranet, a local areanetwork (“LAN”), a wide area network (“WAN”), a virtual private network(“VPN”), a server network, a cellular network, a satellite network, abroadcasting network, a wireless or wired connection, and/or any knownor future network or connection and/or components thereof. The user orclient device 409 is in communication with the server 405 via thenetwork 413.

The user device 409 is configured to provide map, navigation, image,video, and/or geographic related functions or features. For example, theuser device 409 is a desktop computer, a laptop computer, a digitaltablet, a cellular telephone, a mobile phone, a personal digitalassistant (“PDA”), an in-vehicle navigation system, a personalnavigation device (“PND”), a portable navigation device, a watch, acamera (or device that includes camera components), and/or other deviceor computer. The user device 409 includes, but is not limited to, adisplay 433, a processor 437, a memory 441, and one or more inputdevices 445. More, fewer, or different components may be provided. Forexample, transmitter and/or receiver components, audio components,and/or camera components may be provided.

In one embodiment, an end user uses the device 409 to view one or moreimage/video views, such as the image view 300 or other image views, likedifferent street views or streetside views. The processor 437 is similarto or different than the processor 209, 140, or 417, and the memory 441is similar to or different than the memory 213, 144, or 421. Theprocessor 437 is configured or adapted to control and/or communicatewith other components, such as the memory 441, the display 433, and/orthe input device 445, which is similar to or different than the inputdevice 225. The processor 437 processes received data from the server405 to provide features on the display 433, such as photo and/orpanoramic image views including imagery representations as well as maprepresentations. Alternatively, the device 409 does not communicate withthe server 405 to retrieve data to display some or all maps and/orimages, and, instead, the device 409 includes a database, such as thedatabase 425 or portions thereof, stored in a memory, like the memory441, to retrieve such data.

The display 433 is any mechanical and/or electronic display positionedfor accessible viewing in, on, or in communication with the device 409.For example, the display 433 is a touch screen, liquid crystal display(“LCD”), cathode ray tube (“CRT”) display, a light emitting diode(“LED”) display, a plasma display, and/or other display. The display 433may act as a hardware and/or software implementation of a user interfaceor a portion thereof.

FIG. 5 illustrates a graphics user interface 504 on a device 500. Thedevice 500 may be similar to or different than the device 409. Thegraphics user interface 504 may be a software implementation and/orhardware implementation of a display, such as the display 433. Forexample, a photo and/or panoramic image view 508 is displayed. The imageview may be similar to or correspond to the image view 300.

In one embodiment, the photo image view 508 includes an imagerepresentation 516 that represents a street or road, an imagerepresentation 520 that represents a sidewalk or other path, an imagerepresentation 524 that represents a building or other POI, and/or otherimage representations or portions thereof. The image representations maycorrespond to the image data section 304.

The photo image view 508 also includes a map or map widget 512. The map512 corresponds to the region 308. In one embodiment, the map 512 is aninteractive digital map that corresponds to the geographic location ofthe image(s) displayed. Alternatively, the map 512 is a static map. Themap 512 represents the area around the present location that isdisplayed as photo imagery. For example, the icon 528 represents thepoint in the map where one is viewing the images of the real world, suchas the image representations 516, 520, and 524. The icon 528 may be apoint, a circle, an arrow, or any other shape. For example, the roadimage 516 corresponds to the view of Main St. from the location of icon528 in a north direction. Also, the POI 532 corresponds to the imagerepresentation 524. The map 512 includes a graphical model of roads(such as Fake Rd., Green St., Blue Ave., Airport Rd., 2^(nd) St., MainSt., 1^(st) St., Maple Ave, and/or other roads or paths), POIs (such asan airport 556, a park 552, the building 532, and other POIs 536), andother geographic or map features. For example, the map 512 is notphoto/video imagery data but is a vector-based, tile-based, or othertype of graphical map model or representation. For example, the roads inthe map 512 are displayed based on map data, such as a road segment 540and corresponding nodes 544 and 548, that are created by a mapdeveloper, such as the map developer 118. Graphical representations ofthe map are generated and/or displayed based on such map data.

A control input 560 may also be included. For example, the control inputis a software and/or hardware implementation. In one embodiment, thecontrol input 560 is displayed over the photo image view 508 and/or isdisplayed in a separate window. The control input 560 includes, but isnot limited to, a direction compass 564 and zoom buttons 568. More,fewer, or different components or features may be provided. For example,a user is able to pan the photo image view 508 left or right, back orforward, and/or in any other direction using the compass 564 and mayzoom in or zoom out using the zoom buttons 568 (such as via anelectronic mouse or one's finger in regards to a touch screen). Or, thecompass 564 and/or the zoom buttons 568 may not be used in the casewhere one can use a touch screen to move about or zoom or use electronicmouse commands for the same effect.

In one embodiment, the map 512 is integrated with the photo imagery andis part of the photo image view 508. For example, if a user pansupwards, less and less of the map 512 will be displayed, and there is apoint where if the user pans upwards enough, the map 512 will beentirely hidden. Accordingly, the map 512 moves as the photo imagery ismoved. If the photo imagery is rotated, then the map 512 is rotated asif it is part of the photo imagery. For example, if the photo imagery ofthe image view 508 is rotated to place the image representation 524 infront of the users point of view, then the map 512 will integrallyrotate to show the POI icon 532 north of the icon 528 (also, other partsof the map 512 may be exposed that were not exposed via the displayprior to rotation). Alternatively, the rotation or movement in the photoimagery section of the image view 508 may be independent to rotation ormovement in the map 512.

The map 512 may be zoomed in on or zoomed out from without causing anychange in the photo imagery section of the image view 508.Alternatively, zooming operations in the map 512 may cause similar orassociated zooming in or zooming out in the photo imagery section. Also,causing different image views to be displayed from different locations,such as moving forward in the photo imagery, causes the map 512 tochange. For example, a user may want to view the photo imagery down thestreet from where the current photo image view is located, and so theuser navigates forward in the photo imagery. Such an action causes theicon 528 to move forward, such as to the intersection of Fake Rd. andMain St., and sections of the map 512 that were not shown now may beshown (for example, the next street or area that is north of Fake Rd.).Also, the opposite may apply. For example, if the icon 528 is moved inthe map, such as to the intersection of Fake Rd. and Main St., or theintersection of Fake Rd. and Main St. is selected, the photo image viewdisplayed is changed to correspond to the new location.

The map 512 may also provide other navigation-related functions. Forexample, the map 512 may provide routing to different places in the map512 including a display of a generated route. Also, turn-by-turndirections may be provided in visual (including text) and/or audioformats. Also, POI searching or any other searches may be provided tofind something in the map. Alternatively, the map 512 may be a staticmap that only provides for a visual map display.

Only a portion of the photo imagery of the photo image view 508 may bedisplayed at a given time based on the dimensions, size, and/or shape ofthe display or graphics user interface 504 of the device 500. Forexample, when one views the photo image view 508 via the graphics usersinterface 504 at a given time, one may see photo imagery of the photoimage view 508. The photo imagery may be one or more panoramic images orany other type of image. For example, the photo imagery including imagerepresentations 516, 520, and/or 524 may be a panoramic image. Thepanoramic image may be a photo image with an elongated field of view,such as a field of view greater than the human eye from a respectivepoint. For example, the field of view may be at least or greater than180 degrees (such as in the horizontal direction) or may be about orgreater than 160 degrees in the horizontal direction and 75 degrees inthe vertical direction, and/or the panoramic image may have an aspectratio of about 2:1 or larger.

In one embodiment, the photo image view 508 including the map 512 is notlimited to what is displayed or shown at one moment in time. Forexample, the photo image view 508 or 300 corresponds to a 360 degreespherical or bubble view in which different portions are displayed orshown at different times. Alternatively, the photo image view 508 or 300may correspond to a view that is less than 360 degrees and/or is greaterthan 180 degrees. FIG. 6 illustrates a view diagram 601. For example, apoint 609 represents an eye view of a user, such as what a user seesfrom a specific location at a specific time. The 3D bubble or sphere 605illustrates the constraints or dimensions of a photo/panoramic imageview from the specific location. The user views a view plane (made up ofan image view section 621 and a map view section 625) via a view cone613.

For example, the photo image view 508 corresponds to what the user seesbased on the eye or point 609. The image view section 621 corresponds tothe photo imagery representations of the photo image view 508, and theparallel map view section 625 corresponds to the map 512 of the photoimage view 508. Accordingly, the map 512 or portions displayed thereofare in a perspective view that is parallel to the angle of view of thephoto imagery. Alternatively, the map, such as the map 512 or portionsthereof, may be displayed from a top view or a bird's eye view. Forexample, the portion of the map 512 that is being displayed based on theview cone 613 may be displayed in a map section 617 that corresponds toa bird's eye view vantage plane. A section 629 corresponds to theremaining or hidden parts of the map 512 from the bird's eye viewpoint-of-view.

Accordingly, as a user views different parts of the photo image view,like the photo image view 508, the view cone 613 moves accordinglyshowing parts of the photo image view on the display and hiding otherparts. In one embodiment, the view cone 613 may be moved upwards whereinthe map portion of the photo image view is not displayed, even though itexists. The map view may correspond to the bottom half of the sphere605. For example, the map view or map may correspond to at most thebottom 25 percent of the sphere 605. Alternatively, the map view or mapmay be positioned or placed in any other part of the sphere 605, such asin the top half or at most within the top 25 percent. The placement ofthe map view or map may be based on unfocused, blurry, or blockedregions of photo imagery data or ground regions.

FIG. 7 is a flowchart of an exemplary method 700 of generating and/orproviding map, geographic, image, and/or navigation data. Fewer, more,or different steps or acts may be provided, and a combination of stepsmay be provided. Also, the steps or acts may be performed in the orderas shown or in a different order. The method is implemented by thesystem and/or devices described herein or by different devices orsystems. One or more steps or processes of the method may be fully orpartially automated (such as via a computer, processor, and/oralgorithm).

In one embodiment, data is collected (Step 704). A map developer employsor gets people to collect geographic, image/video, and/or other data,such as via the object 108 and/or the device 112, about one or moreregions or areas. For example, roads or streets are driven with avehicle having one or more cameras to collect surrounding imagery/video,location data, and/or other geographic data. The collected data, such asdata in the media 116, is sent to a map developer facility, such as themap developer 118, or equipment thereof for processing.

Composite, stitched, and/or panoramic images or other images, such asthe image data 304, as well as map data, such as the map data 312, aregenerated or provided (Step 708). For example, collected imagery data isprocessed to generate (or converted to) panoramic images of geographicareas, such as panoramic street views or journey views, using known orfuture image processing techniques. Further description and detailsabout stitching and/or composite images may be provided in the U.S. Pat.No. 7,688,229, entitled SYSTEM AND METHOD FOR STITCHING OF VIDEO FORROUTES, which is hereby incorporated by reference. These panoramicimages are linked or associated with corresponding maps or map data,such as the map 512, to generate photo/panoramic image views, such asthe image view 300 or 508. In one embodiment, the map widget or maprepresentation is integrated within the photo imagery, such as from thedisplay point-of-view (as opposed to being in a separate window and/orjust hovering over the photo imagery). The map may be placed orintegrated in any portion of the photo imagery. For example, the map maybe placed in a region corresponding to a fuzzy, unfocused, blurry, orblocked region (e.g., an unfocused, blurry, or blocked region iscompletely covered by the map, and/or the map only covers an unfocused,blurry, or blocked area in the imagery). Or, the map may be placed in aregion corresponding to a ground area (e.g., the map is integrated inonly area(s) that correspond to a ground in the imagery) or other area,such as a sky area.

The processed data, such as generated photo image views and/or data ordata links thereof are stored (Step 712). For example, camera image dataand map data that correspond to a photo image view, such as the photoimage view 300 or 508, may be stored together in one or more data files.Or, the photo image data may be stored separately from the map data, andthe links or data associations that tie the specific photo image data tospecific map data or representation(s) may be stored. The data is storedusing known or future storing techniques in one or more databases, suchas the database 128, 132, 136, and/or 184. Then the generated, stored,and/or processed data corresponding to the photo/panoramic image views(including map data and/or links to map data) are sent or provided toother entities, such as end user applications/devices orgeographic/map/navigation service/device providers (Step 716). In oneembodiment, some or all of the data may be hosted on a server, such asthe server 405, and/or provided in end user devices, such as the device409.

FIG. 8 is a flowchart of an exemplary method 801 of operating a userdevice, such as the device 409 or 500, or application thereof. Fewer,more, or different steps or acts may be provided, and a combination ofsteps may be provided. Also, the steps or acts may be performed in theorder as shown or in a different order. The method is implemented by thesystem and/or devices described herein or by different devices orsystems. One or more steps or processes of the method may be fully orpartially automated (such as via a computer, processor, and/oralgorithm).

In one embodiment, a user turns on his or her computer, digital phone,or other device and goes to an Internet site or opens an application.The user is able to view geographic scenes, such as real world imagestreet views. The user enters or inputs location information, such as anaddress or a name of a city or street (or, an area or location isclicked on). The computer, phone, or other device, such as the device409 or 500, receives the user entered input (Step 805). Based on theinput, the device retrieves data of one or more photo image viewscorresponding to the input (Step 809). For example, data of the photoimage view 508 (including image data 516, 520, and 524 as well as map512) is retrieved from a remote database, such as the database 425, orfrom data stored on the user device. One or more photo/panoramic imageviews and/or data thereof are provided to the user device, such as fordisplay (Step 813). For example, the photo image view 508 is displayedto the user (Step 817).

The user is able to interact with the photo image view. For example, theuser zooms in on or out from, rotates, pans, and/or navigates within themap portion or imagery portion of the photo image view. The user deviceor application thereof receives control input based on the user'sselected actions (Step 821). Based on the control input, the display ischanged to correspond to the user's selection(s). For example, theintegrated map widget rotates or pans with the rotation of the imageryportion of the panoramic image view. A user or viewer icon, such as theicon 528, may be provided in the map section and change or move based onuser selection and/or different navigation within the imagery or map.The map section may be zoomed in on or zoomed out from independentlyfrom the imagery section of the photo image view. Different features orfunctions of the photo image view discussed herein may also be provided.Alternatively, the map section may be a static map that is notinteractive.

Other or further functions via the photo image view and/or applicationthereof may be provided (Step 825). For example, navigation routing froman origin point to a destination point may be provided, such as usingknown or future routing techniques (e.g., a Dijkstra method, an A-staralgorithm or search, and/or other route exploration or calculationalgorithms may be used). For example, the map section may be a navigablemap that includes navigation-related functions including routecalculation and map guidance. A generated route may be displayed on orover the map section to assist a user. Turn-by-turn routing directionsmay be provided in visual and/or audio format.

For example, a video showing the way from one point to another point maybe provided (such as a preview or demonstration), and while thedifferent photo imagery of different photo image views are beingpresented as a video, the map section may show a moving icon(corresponding to the movement of the photo images) along roads or pathsin the map. Other navigation or geographic related functions, such asPOI searching functions, may be provided. For example, if a POI or areaof interest is searched for, the map section may highlight that POI orarea within the map widget. Further description and details aboutnavigation-related functions may be provided in the U.S. Pat. No.6,278,942, entitled METHOD AND SYSTEM FOR PROVIDING ROUTING GUIDANCE,which is hereby incorporated by reference.

The features, components, devices, systems, acts, steps, and/orprocesses mentioned herein may be mixed and matched to provide othersystems and/or methods embodying the idea(s) disclosed.

Alternatives

The embodiments of the photo/panoramic image view focus on a street viewor outdoor view. One or more generated photo image views may be indoorpanoramic/photo image views. FIG. 9 illustrates an indoor photo imageview 900, such as via a graphics user interface, like the graphics userinterface 504. The indoor photo image view 900 may have similar ordifferent functions/features as the photo image view 508.

In one embodiment, the image view 900 includes a photo imagery sectionincluding image representations 904 (representing walls of a corridor inan indoor facility), an image representation 908 (representing abuilding floor), an image representation 914 (representing an indoorceiling), an image representation 912 (representing a door), and/orother image representations. The image representations are part ofindoor photo imagery collected and processed, such as the image data 304or imagery of the photo image view 508. A map or map widget 920 (whichmay be similar to the map 512) is provided as an integral part of thephoto imagery. The map 920 is a layout map or a floor plan of the indoorspace and is provided in the ceiling 914. The map 920 is provided in theceiling so as not to disturb any floor features or other features in thephoto imagery. The map 920 may be provided in other parts of the photoimagery, such as based on the different considerations mentioned herein.Alternatively, the map 920 may be provided independent of thephoto/video imagery, such as in a separate window. The photo imagery maybe panoramic imagery or other types of imagery. The map 920 may be anode/segment map, an open area map, or other map. Further descriptionand details about open area maps may be provided in the United StatesPatent Application Publication, U.S. 2010/0023250, entitled OPEN AREAMAPS, which is hereby incorporated by reference.

In one embodiment, the integration and/or relationship between the mapand the imagery is similar to the diagram 601. For example, an icon orpoint 928 represents a location where the user is viewing the imagery,and an icon 924 corresponds to the door 912. A wall representation 936corresponds to the right side wall 904, and a wall representation 932corresponds to the left side wall 904. Accordingly, the map or floorplan 920 is represented as a top mirror view. The map 920 may bedisplayed from a parallel perspective view relative to the imagery viewpoint or may be displayed from a straight up view (or, alternatively,from a bird's eye view (a straight down view)).

The photo image view 900 (including the map 920 and the photo imagery)may include or provide similar features or functions as the photo image508, such as movement, rotation, panning, zooming, interactivefunctions, navigation-related functions (such as routing and/or routedisplay), map display functions, search functions, and/or otherfunctions. In one embodiment, using the map 920, a route from theposition 928 to an office is generated, and a route representation(s)940 is provided for guidance purposes. Also, a control input, such asthe control input 560, or other control input features may be used.Method steps or features similar to those described herein forgenerating, providing, and/or using one or more photo image views, suchas the methods 700 and/or 801, may apply and correspond to the indoorphoto image view 900. For example, indoor location or area input isreceived, and photo image data and map data corresponding to the indoorlocation or area is retrieved to display and enable an indoor photoimage view, such as the image view 900.

In one embodiment, virtual lighting may be used to highlight an objectpresented in the photo imagery. For example, if a user selects the door912 or 924 (or a route is generated through the door 912 or 924), avirtual overhead light 916 provides lighting or highlighting 918 on,above, over, and/or by the image representation 912. Accordingly, such avirtual light acts as guidance for the user in the photo imagery sectionof the panoramic image view 900. The virtual lighting or light(s) may beprovided from any other direction or point in the image view 900.

The logic, data, software, or instructions for implementing the systems,processes, methods and/or techniques discussed above are provided oncomputer-readable storage media or memories or other tangible media,such as a cache, buffer, RAM, removable media, hard drive, othercomputer readable storage media, or any other tangible media. Thetangible media include various types of volatile and nonvolatile storagemedia. The functions, acts, steps, or tasks illustrated in the figuresor described herein are executed in response to one or more sets oflogic or computer-executable instructions stored in or on computerreadable storage media (such as non-transitory media). The functions,acts or tasks may be independent of the particular type of instructionsset, storage media, processor or processing strategy and may beperformed by software, hardware, integrated circuits, firmware, microcode and the like, operating alone or in combination. Likewise,processing strategies may include multiprocessing, multitasking,parallel processing and the like. In one embodiment, the instructionsare stored on a removable media device for reading by local or remotesystems. In other embodiments, the logic or instructions are stored in aremote location for transfer through a computer network or overtelephone lines. In yet other embodiments, the logic or instructions arestored within a given computer, central processing unit (“CPU”),graphics processing unit (“GPU”) or system.

It is intended that the foregoing detailed description be regarded asillustrative rather than limiting and that it is understood that thefollowing claims including all equivalents are intended to define thescope of the invention.

I claim:
 1. A computer-implemented method of providingnavigation-related functions including a map display, the methodcomprising: receiving input corresponding to a geographic location;retrieving photo image data associated with the geographic location, thephoto image data based on at least one image captured by a video cameraor a photo camera; providing the photo image data to display at least aportion of a photo image view representing at least a portion of thegeographic location; retrieving geographic map data corresponding to thegeographic location; and providing the geographic map data to display atleast a portion of a digital map in the portion of the photo image view,wherein the digital map is integrated within the photo image view,wherein the photo image view comprises a panoramic 360 degree viewaround a geographical point.
 2. The method of claim 1, wherein thedigital map is integrated in a region corresponding to a ground orfloor.
 3. The method of claim 1, wherein the digital map is integratedin an area associated with a blurred or unfocused region.
 4. The methodof claim 1, wherein the digital map is a graphical model of at least aportion of the geographical location generated from the geographic mapdata.
 5. The method of claim 4, wherein the geographic map data includesroad segment data records.
 6. The method of claim 1, wherein the digitalmap is a graphical model of at least a portion of the geographicallocation from a bird's eye view or a perspective view.
 7. The method ofclaim 6, wherein the perspective view is parallel to a view plane ofimagery of the photo image view.
 8. A non-transitory computer-readablemedium configured to store a computer program that performs a method ofoperating a navigation system, the method comprising: receiving inputindicating an area; obtaining photo image data associated with the area,the photo image data based on at least one image captured by a videocamera or a photo camera; displaying at least a portion of a photo imageview representing at least a portion of the area based on the obtainedphoto image data, wherein the photo image view comprises a street viewor an indoor view; obtaining map data corresponding to the area; anddisplaying at least a portion of a digital map as a function of theobtained map data, the digital map integrated in the photo image view.9. The non-transitory computer-readable medium of claim 8, wherein thephoto image view comprises a panoramic view with a field of view greaterthan 160 degrees.
 10. A non-transitory computer-readable mediumconfigured to store a computer program that performs a method ofoperating a navigation system, the method comprising: receiving inputindicating an area; obtaining photo image data associated with the area,the photo image data based on at least one image captured by a videocamera or a photo camera; displaying at least a portion of a photo imageview representing at least a portion of the area based on the obtainedphoto image data; obtaining map data corresponding to the area; anddisplaying at least a portion of a digital map as a function of theobtained map data, the digital map integrated in the photo image view,wherein the digital map is integrated in a region associated with aground area, an unfocused area, or a ceiling area of the photo imageview.
 11. The non-transitory computer-readable medium of claim 8,wherein the digital map is configured to provide navigation-relatedfunctions including turn-by-turn routing instructions or guidance. 12.The non-transitory computer-readable medium of claim 8, wherein thedigital map comprises an interactive digital map configured for panning,rotating, or zooming operations.
 13. The non-transitorycomputer-readable medium of claim 8, wherein the digital map comprisesan interactive digital map configured to change the photo image view toanother photo image view by selecting a point on the interactive digitalmap.
 14. The non-transitory computer-readable medium of claim 8, whereinthe digital map rotates with imagery of the photo image view.
 15. Thenon-transitory computer-readable medium of claim 8, wherein the digitalmap corresponds to an indoor floor plan.
 16. A navigation systemcomprising: a graphical user interface display; and a processor incommunication with the graphical user interface display, the processorconfigured to retrieve panoramic image data and geographic map dataassociated with a geographic area based on an input indicating thegeographic area, wherein at least a portion of a panoramic image viewrepresenting at least a portion of the geographic area is displayed onthe graphics user interface display based on the panoramic image dataprovided by the processor, and wherein at least a portion of a digitalmap is displayed on the graphics user interface display as a function ofthe geographic map data provided by the processor, the digital map beingintegrated into the panoramic image view, wherein the digital mapincludes an icon positioned at a geographic point corresponding to thepanoramic image view.
 17. The navigation system of claim 16, wherein thedigital map is a graphical map widget representing at least a portion ofthe geographical area from a bird's eye viewpoint or a perspectiveviewpoint.
 18. The navigation system of claim 16, wherein the panoramicimage view comprises an interactive panoramic image view configured tobe panned to expose the different portions of the panoramic image viewon the graphics user interface display, wherein panning of imagery ofthe panoramic image view causes the integrated digital map to pan in thesame direction.
 19. The navigation system of claim 16, wherein theintegrated digital map comprises an interactive digital map configuredto be panned, and wherein panning of the interactive digital map causesimagery of the panoramic image view to pan in the same direction. 20.The navigation system of claim 16, wherein the panoramic image viewcomprises an interactive panoramic image view configured to be zoomed inon or zoomed out from, and wherein when imagery of the panoramic imageview is zoomed in on or zoomed out from, a different panoramic imageview is displayed on the graphics user interface display relative to anew geographical point corresponding to the zooming operation.
 21. Thenon-transitory computer-readable medium of claim 8, wherein the photoimage view comprises a panoramic view of at least 180 degrees around ageographical point.
 22. The non-transitory computer-readable medium ofclaim 8, wherein the digital map includes an icon positioned at ageographic point corresponding to the panoramic image view.
 23. Thenon-transitory computer-readable medium of claim 8, wherein the photoimage view comprises an interactive panoramic image view configured tobe panned to expose different portions of the photo image view.